COXETER GROUPS, 2-COMPLETION, PERIMETER 
REDUCTION AND SUBGROUP SEPARABILITY 

Paul E. Schupp 

Abstract. We show that all groups in a very large class of Coxeter groups are locally 
quasiconvex and have uniform membership problem solvable in quadratic time. If a 
group in the class satisfies a further hypothesis it is subgroup separable and relevant 
homomorphisms are also calculable in quadratic time. The algorithm also decides if 
a finitely generated subgroup has finite index. 



§1. Introduction. 

Several years ago the author [S2] raised the question of whether or not smaU 
canceUation methods could be used to investigate questions about finitely gener- 
ated subgroups of "sufficiently nice" groups — in particular the solvability of the 
membership problem and the Howson property. Rips [R] replied "No" by con- 
structing, for every metric small cancellation condition C (:^), finitely presented 
groups which satisfy the condition but which have unsolvable membership prob- 
lem and are neither Howson nor coherent. However, recent work of McCammond 
and Wise [M-W] and of Arzhantseva and 01shanskii[A-0] shows that the answer 
is actually "Yes" in many cases. McCammond and Wise introduced the use of 
"distributive" small cancellation hypotheses where the condition involves how the 
generators are distributed among all the defining relators and the ingenious idea of 
"perimeter reduction" where one counts "what is missing" . 

In this paper we introduce the idea that if one has a "suitable" subgroup graph, 
Ai(iJ), of a subgroup H, one can construct from it a "complete" subgroup graph, 
^2{H) which has most of the properties of subgroup graphs in the case of free 
groups and which directly reveals desired information about H. . Thus there are 
two distinct stages. The first is to show that finitely generated subgroups indeed 
have "suitable" subgroup graphs. In this paper we show that for a very large class 
of Coxeter groups, and for surface groups, one can start with the usual subgroup 
graph Ao(-ff) for an arbitrary finitely generated subgroup H and use perimeter 
reduction to obtain a subgroup graph /S.i{H) which satisfies a certain "Relator 
Path Property" , which is a precise definition of "suitable" . 
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We then show that any subgroup graph whatsoever with the Relator Path Prop- 
erty can be completed to its "2-complction," A2{H). A "2-complete" subgroup 
graph has the desired strong properties. Indeed, one can construct the 2-completion 
of any graph ( not necessarily a subgroup graph), having the Relator Path Property 
and we shall use this fact later. Also, Kapovich and Schupp[K-S] show that the 
Arzhantseva-Olshanskii technique mentioned above, which is quite different from 
perimeter reduction, also constructs subgroup graphs with the Relator Path Prop- 
erty. 

The graph A2{H) describes the "Dehn hull" of H. That is, any Dehn reduced 
word w ( one not containing more than half of a defining relator) represents an 
element of H if and only if w is the label on a closed path at the basepoint of 
A2{H) . The graph A2{H) is the transition graph of a finite automaton which 
accepts a Dehn reduced word w if and only if w & H. Since any geodesic word 
which is Dehn reduced, this immediately shows that H is rational and quasiconvex 
and that the diameter of A2{H) is a quasiconvexity constant for H. Next, the 
subgroup H has finite index if and only if the graph A2{H) is full, that is, there 
are edges labelled by all generators incident at all vertices. 

Perhaps most interestingly, in the case of Coxeter groups, if G satisfies an addi- 
tional "Separability Condition " then G is subgroup separable and, given w ^ H, 
by slightly modifying A2{H) to A2{H,w), one can directly read off from the lat- 
ter graph a honioniorphism 99 from G into a finite symmetric group such that 
(p{w) ^ '^{H). Indeed, there is a uniform quadratic time algorithm calculating all 
the above information. 

A Coxeter group G is a group with a finite presentation 

G= (ai,...,an; a-, (aiaj)""*^ , i ^ j) 

where the rriij > 1 and we may have niij — 00, signifying the absence of a relation 
between and aj . The Coxeter matrix of the presentation is the symmetric matrix 
M = (rriij) where each ma = 1. The modified Coxeter graph T of the presentation 
is the graph with vertices ai, . . . , a„ and, if rriij < 00, there is an edge between 
and ttj labelled by rriij. Note that our modified graph differs from the usual Coxeter 
graph in that we omit edges labelled by 00 and include edges labelled by 2. Thus 
the absence of an edge between and aj means that rriij = 00 When we say that 
we are "given" a Coxeter group, we always mean that we are given such a finite 
presentation. 

In general, Coxeter groups may be very bad with respect to the properties in 
which we are interested. Let i? = F2 x F2 be the direct product of two free groups of 
rank 2. Mikhailova's Theorem [see L-S] shows that there is a fixed finitely generated 
subgroup H of B such that the membership problem for H in B is unsolvable. 
Furthermore, B is neither Howson nor coherent. The class of Coxeter groups is 
closed under both free products and direct products and the free product K of 
three cyclic groups of order two contains free subgroups of rank two. Thus K x K 
is a six-generator right-angled Coxeter group containing a copy of B and so has 
unsolvable membership problem and is neither Howson nor coherent. 
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We turn to formulating a suitable Reduction Hypothesis which will define the 
class of Coxeter groups to which our results apply. First of all, we assume that 
there are at least three generators and that all niij > 4. In the terminology of 
Appel and Schupp [A-S], we only consider groups of extra-large type. Such groups 
satisfy the small cancellation hypothesis C" (y) and thus are hyperbolic in the sense 
of Gromov. (The only use we make of small cancellation theory is in the statement 
of two facts.) 

The set A of generators is ordered 

ai < a2 < ■ ■ ■ < ttn 

by increasing subscript. This induces the shortlex or canonical ordering in the set 
A* of all finite words on the generators. Write \u\ for the length of u and define 
'U<t'if|tt|< |f|orif|ti| = |t;| and u precedes v in the lexicographical ordering on 
A* induced by the ordering on A. The normal form of an element of G is the 
least element u in the shortlex ordering such that u = g in G. 

Tits [T] proved that for any Coxeter group, an arbitrary word can be transformed 
into normal form by a finite sequence of operations: 

(I) Cancel an occurrence of a generator and its inverse (which is just the gen- 
erator in the case of Coxeter groups). 
(II) Replace an occurrence of half a defining relator by the inverse of the other 
half. 

Since we also want to state results about surface groups at the end of this article, 
we point out that the same result holds for suitable small cancellation groups. 

Normal Form Theorem. Let G = {X: R) be a finitely presented group in which 
the symmetrized set R generated by R satisfies the small cancellation G' (y), has all 
pieces of length 1 and has only elements of even length. Then the set N of shortlex 
normal forms of elements of G is a regular language and there is a linear time 
algorithm which, when given an arbitrary word w calculates the normal form ofw. 
A word w can be transformed into normal form by a finite sequence of operations I 
and II. 

This theorem is essentially now well-known but we include a proof along with 
the discussion of Dehn convexity in the Appendix. 

We want to study finitely generated subgroups by constructing their subgroup 
graphs exactly as in the case of a free group. Given a Coxeter group G = {A;R), 
a graph over G is a graph in which edges are labelled by generators from A. If 
the subgroup H is generated by {hi, ... , hm} we start with a bouquet of m loops 
arranged around a basepoint Oh- The i-th loop is then subdivided into edges 
labelled by generators so that one reads hi counterclockwise around the loop. (The 
hi are arbitrary words, not necessarily in normal form.) Call this graph Ao{H). 

3 



Figure 1 



Since the generators have order two, we read the same label in both directions 
along an edge. If two edges with the same label a are incident at a vertex v we fold 
(identify) the two edges into a single edge. 

Folding is the only operation on the graph which is needed in the case of a free 
group and takes care of the relations a| = 1 in the case of Coxeter groups, but we 
now need to deal with the other relations. Two generators and aj are related if 
rriij < oo, that is, there is a relator {aiaj)^^K We view relators as cyclic words. 
Since the generators have order 2, a relator and its inverse are the same cyclic word. 
For each i, let pi denote the number of other generators Oj, j ^ i, to which Oj is 
related. Set pij = max{pj,pj}. We shall see later that the following hypothesis is 
sufficient to use McCammond-Wise perimeter reduction to construct Ai{H). 

The Reduction Hypothesis.. G — {A; R) is of extra-large type and there is a 
subset C C. A such that every defining relator {aiaj)'^''^ contains a generator from 
C and satisfies the following condition. 
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(1) // both tti and aj are in C then rriij > -pij, 

(2) If tti & C and aj ^ C then rriij > '^Pi- 

One can always, of course, make the "uniform choice" C = A and then the 
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hypothesis is that all mij > -^Pij- 

The Reduction Hypothesis is best viewed as a condition on the modified Coxeter 
graph r of G. Since the vertices Vi correspond to the generators Oj, pi is the degree 
of the vertex vi. Since rriij < oo labels an edge e^j between Vi and Vj, pij is the 
maximum of the degrees of the endpoints of e^-. A vertex cover of the graph T 
is a set C of vertices such that every edge has at least one endpoint in C. The 
Hypothesis is that rriij > \pij if both endpoints of Cij are in C and rriij > 2pi if 
only one endpoint Vi G C. For example, suppose that F is a central triangulation of 
a polygon, say with the central vertex. In this case, taking C = {ai, . . . , an-i} 
imposes the condition rriij > 5 on the outer edges and the condition rrini > 7 on 
the central edges, and this condition is independent of n. In general, the condition 
on rriij depends only on degrees of the endpoints of the edge e^j. 

If we used instead of oo in the Coxeter matrix to represent the absence of a 
relation, then having a large number of missing relators is a sparseness condition 
on the matrix. A sparse matrix allows relatively small values for the rriij. 
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Figure 2 

The additional hypothesis which we need to estabhsh the Coxeter group G is 
subgroup separable is easily stated. Why it is needed will be clear when we discuss 
the structure of the graphs /\2{H). 

The Separability Condition.. All rriij are even and if rriij is the label on an 

edge which forms part of a triangle in the modified Coxeter graph T then ruij is also 
divisible by 3 (and is therefore divisible by 6). 

Note that if the Coxeter graph F does not contain any triangles then the hypoth- 
esis needed for subgroup separability is the Reduction Hypothesis plus having all 
niij even. 

We can now state our main theorem. 

Theorem I. Then there is a fixed quadratic time algorithm which, when given a 
Coxeter group G = (A; R) satisfying the Reduction Hypothesis, a tuple {hi, . . . , hm) 
of generators for a subgroup H and an element w of G: 

(1) Calculates the graph A2{H) which, in particular, is the graph of a finite 
state automaton which accepts a Dehn reduced word u if and only ifu&H. 

(2) Calculates a quasiconvexity constant for H . 

(3) Decides if H has finite index. 

(4) Decides ifw&H. 

(5) Ifw^H and G satisfies the Separability Condition, the algorithm explicitly 
writes out a homomorphism (p from G to a finite symmetric group such that 
(p{w) ^ip{H). 

Before proceeding with the proof we make a few comments. (See also Gersten 
and Short [G-S] who pointed out the close connection between rationality and qua- 
siconvexity.) Since we have fixed shortlex normal forms, we say that a subgroup H 
is rational if the set of normal forms of elements in is a regular language. Part 
(1) of the theorem shows that all finitely generated subgroups are rational. This 
property directly shows that G is Howson by an argument which is now well-known. 
If H and K have graphs A2{H) and A2{K) respectively, then the component of 
A2{H) X A2{K) which contains [Oh, Ok) is the transition graph of a finite au- 
tomaton which accepts a word z in normal form if and only if^ei^ni^, soi/flK 
is also rational. The Anisimov-Seifert theorem shows that any rational subgroup is 
finitely generated. 
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A subgroup H is a. quasiconvex if there exists a constant d such that every point 
on a geodesic word which represents an element of H is within distance d of an 
element of H. Since a geodesic word is Dehn reduced, the graph A2{H) is convex 
in the sense that any geodesic word representing an element of H is the label on a 
path is A2{H) which begins and ends at the basepoint Oh- Define the diameter d 
of A2{H) to be the maximum distance of any vertex in the graph to the basepoint. 
It follows that (i is a quasiconvexity constant for H. Furthermore, the well-known 
Dijkstra Distance Algorithm runs in quadratic time and labels each vertex of A2{H) 
by its distance from Oh, so d is calculable in quadratic time. 

The size of the graphs A2{H) is "uniformly linearly bounded." Let 



be the sum of the lengths of the given generators of H. Let ko be the maximum 
length of any relator times the maximum of the pi (the number of relators involving 
Gi). The number of edges at any stage of the construction of A2{H) will never 
exceed ko sh- We have implemented the calculation of A2{H) with a computer 
program and for subgroups with sh around 1500 the calculation takes only a few 
seconds. Indeed, our ideas about the theorem have been formed to a great degree 
by computer experiments. The program will hopefully be available soon on the 
author's webpage. 



To prove the theorem we now describe precisely the construction of the graph 
A2{H) in two phases. Given generators hi, ... , hm for if, we begin with the stan- 
dard graph Aq[H) which is a bouquet of m loops at a basepoint Oh where the i-th 
loop is labelled by hi. The number of edges in Aq{H) is exactly sh- 

A closed path in the subgroup graph which is labelled by a defining relator is 
called a relator cycle. If we see a path labelled by "enough" of a defining relator 
but which is not part of a relator cycle, we want to add the missing part of the 
relator to form a relator cycle. This enlarges the graph and we need the Reduction 
Hypothesis to guarantee that the process stops. 

Here we use the beautiful idea of McCammond and Wise of perimeter reduction. 
We have chosen a subset C C. A such that every defining relator [aittj)"^^^ contains 
a generator from C. Given a current subgroup graph A, for each edge e labelled 
by a generator Oj G C, we count the number of relator cycles which are missing at 
e. That is, the number of distinct relators (ajaj)"^*^ such that e is not on a relator 
cycle in A labelled by (ajaj)'^*^'. 

The count 7(A) is the sum of four times the number of missing cycles over all 
edges labelled by generators from C plus the current number of edges. For our 
purposes, "enough" of a relator is a relator with at most three letters missing. Our 
Reduction Hypothesis simply ensures that completing such paths to relator cycles 
reduces the count. 



m 




§2. The Construction. 
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Suppose we see a path a in A labelled by a relator (aiUj)^^^ with three letters 
missing and a is not already on a relator cycle labelled by (aiaj)'^^^ . Then we add 
three new edges and two new vertices to the graph so that 0:616263 is a relator cycle. 



There are two cases in considering how this effects the count. First suppose that 
both Gi and aj are in the chosen set C of generators. The cycle {aiaj)'^^^ is missing 
at each of the edges in ck, and we have now filled it in, so we have reduced the 
number of missing cycles by 2my — 3. The new edges are on the cycle we just 
formed so there are at most 3(pjj — 1) missing cycles on the new edges. Since we 
multiply the number of missing cycles by 4 and we have added at most three new 
edges, the condition to reduce the count is just 



which is nriij > \pij- 

The second case is that only one of the generators, say a^, is in C while aj ^ C. If 
two of the new edges are labelled by then completing a to a relator cycle reduces 
the number of missing cycles by mij — 2 and increases the number of missing cycles 
by 2(pj — 1). So the condition in this case is 



or rriij > 2pi. If only one new edge is labelled by and the other two are labelled 
by aj the count is even further reduced. We alternate the following two operations. 

(I) If there are two edges 61 and 62 with the same label Oj incident at a vertex 
f, we fold the two edges. Note that folding reduces the number of missing cycles if 
ai E C unless both 61 and 62 were already on all possible relator cycles involving 
the generator a^. Even in this case or if aj ^ C, folding reduces the number of edges 
so the count is reduced. We continue folding until the current graph is trim, that 
is, there are no pairs of edges to be folded. 

(II) We search the current graph for a path a such that a is labelled by a relator 
with at most three letters missing but a is not part of a relator cycle. We then 
complete a to a relator cycle. Since G satisfies the Reduction Hypothesis this 
operation reduces the count 7. Of course, the resulting graph need not be trim so 
we return to Step I. 




Figure 3 



2mij - 3 > Z{pij - 1) 



ruij - 2 > 2(pi - 1) 
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We iterate (I) and (II) until neither is applicable. Since the original count for 

Ao{H) was at most bkosu and each step reduces the count and performing one 
step requires only searching once through the current graph, the process terminates 
with a graph Ai(i/) in time quadratic in sh- 

For future reference, note that the iteration of Step II ensures that if there is a 
loop labelled by {aiaj)^ at a vertex and rriij < oo then this loop is part of a relator 
cycle so k divides rriij. 

We again stress that we have used the Reduction Hypothesis to ensure that 
Ai(i7) has the following property. 

The Relator Path Property. Any path a labelled by a relator with at most three 
letters missing is part of a relator cycle. 

Our construction of /S.2{H) from Ai {H) depends only on this property and makes 
no further use of the Reduction Hypothesis. In complete generality, given any 
finite graph over G which has the Relator Path Property, we can construct its 2- 
completion, so it is time to give a precise definition of what we mean. Since we also 
want to construct completions in the case of surface groups, we give a definition in 
the case of a group with a small cancellation presentation. 

2-Complete Graphs. Let G =< X;R > be a finitely presented group where R 
satisfies the small cancellation condition C"(l/6) and all pieces have length one. A 
graph T over G is 2-complete if whenever two edges, ei = and 62 = (f ,^2) 

are incident at the vertex v then 6162 is part of a relator cycle except possibly in the 
following situation: Both vi and V2 have degree 2, and, there is an edge 63 = (^3, v) 
incident at v such that both e^e^^ and 636^^ are part of relator cycles. 

A 2-comletion V2 of a graph V is a finite 2-complete graph which contains V as 
an embedded subgraph. 

Since we are stressing the independence of the two phases of our construction, 
we state the following as a theorem. 

Theorem II. Let G be a Coxeter group of extra-large type. There is a uniform 
quadratic time algorithm which, when given any finite graph Ai over G which has 
the Relator Path Property, constructs a 2-completion A2 0/ Ai. 

Proof. We have to complete missing relator cycles. Call the edges of Ai primary 
edges and the vertices of Ai prim,ary vertices. An {i,j)-path in Ai is a path of 
length at least two such that edge labels alternate between Oi and aj. The first 
edge may be labelled by either or aj, so a (j, i)-path is the same thing as an 
(i,j)-path. Now Ai has the property that any maximal (i,j)-path which is not 
part of a relator cycle is a simple path and, since G is of extra-large type, requires 
adding at least four edges to complete it to a relator cycle. 

For every maximal (i, j)-path in Ai, 1 < i < j < n, which is not already part of 
a relator cycle; we add the missing edges to complete that path to a relator cycle. 
New edges and vertices which are added in this process are called secondary. Call 
the resulting graph A^. 
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Consider a primary vertex v in Ai and the maximal paths beginning at v which 
were completed to relator cycles by adding secondary edges. If such a path a is an 
(i, j)-path beginning with an edge labelled by a^, then by the maximality of a there 
is no edge labelled by aj incident at v. There may also be an (£, j)-path beginning 
at V whose first edge is labelled hy a£, £ ^ See the figure below where the edges 
on the top of the figure are primary edges and the edges on the bottom are new 
secondary edges. 



Identify all the secondary edges beginning at v with label aj, making the graph 
A'l trim at the vertex v. Let v' be the secondary endpoint of the secondary j-edge 
incident at v. We call such vertices critical vertices. Since the edges other than the 
J -edge at v' are labelled by the distinct generators Oj, a^, . . . beginning the maximal 
paths starting at v which were completed to relator cycles, the new graph after 
folding the j-edges at v is also trim at v' . 

Hence, identifying new secondary edges with the same label which are incident 
at a primary vertex v over all primary vertices v e Ai results in a trim graph A'/ 
which is completely folded. 

We still need to complete any pairs of edges, say labelled by and at which are 
incident at a primary vertex v, and have nist < oo but are not on an (s, t)-relator 
cycle. The construction above ensured that any two primary edges which meet at a 
primary vertex and are labelled by related generators lie on a relator cycle. But the 
current situation can occur when one or both of the edges of the pair are secondary 
edges. 

First suppose that the edge ei labelled by is primary while the edge 62 labelled 
by at is secondary. Let u be the other endpoint of ei. There could also be a 
secondary edge 63 labelled by at incident at u. If this is the case complete 636162 
to an {s, t)-relator cycle. If not, we complete 6162 to a relator cycle. If 61 and 62 
are both secondary edges, complete 6162 to an (s, t)-rclator cycle. 

The result of having done this for all pairs of edges incident at primary vertices 
is A2. The claim is that A2 is trim as constructed. Take, for instance, the case 
where we complete two edges 61,62 to an (s, t)-relator cycle as in the paragraph 
above. If there is a secondary edge 62 labelled by at incident at a primary vertex v 
then there is no primary edge labelled by at incident at that vertex. Also, 62 was 
introduced in completing some maximal {£, t)-path to a relator cycle. So there is 
an i-edge incident at v. Since this cycle then exists, s ^ I. (There may be several 
choices of £ but none of them are equal to s.) So the new s-edge introduced at the 




Figure 4 
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end of 62 keeps a trim graph. The argument is the same if there is another edge 63 
involved or in the case that both ci and 62 are secondary. 




Figure 5 



Recall that a critical vertex is a secondary vertex v such that there is a secondary 
edge from v to a primary vertex. The graph A2 has the crucial property that: If 

edges labelled by ai and aj meet at any noncritical vertex then either ai and aj are 
not related or the edges lie on an {i, j)-relator cycle. If edges 61,62 labelled by ai 
and aj meet at a critical vertex v and ai and aj are not on an (i, j) -relator cycle 
then there is a secondary edge labelled by some generator at, t ^ i,j incident at v 
and the other endpoints of ei and 62 have degree 2 and the incident edges are also 
labelled by at- 

To establish this last point we use the fact that since G is of extra-large type 
all relator cycles newly introduced have at least four secondary edges. Only the 
secondary edges at each end of a path of secondary edges completing a relator cycle 
can also lie on other relator cycles. So if a secondary edge e labelled by is incident 
to a critical vertex which is the endpoint of an edge labelled by at, the only edge 
incident at the other endpoint of e is labelled by at- That all pairs of edges incident 
at a vertex and labelled by a related generators are on a relator cycle except in the 
one case discussed above is the crucial property of the 2-completion A2). 

We have seen that A2 is trim. Any reduced path labelled by half a defining 
relator lies on a relator cycle. Indeed, the only path of length 2 not on a relator 
cycle is formed by edges labelled by Oj and aj incident at a critical vertex as above 
and this is a maximal {i, j)-path. □ 

Since 2-completion only involves completing relator cycles, it is clear that if we 
start with a subgroup graph Ai (H) with the Relator Path Property, its 2-completion 
A2 (H) contructed above is also a subgroup graph for H. The Normal Form Theorem 
thus shows that if u is the label on a path from a vertex vi to a vertex V2 in A2 (H) 
then there is a path from vi to V2 labelled by the normal form w of u. We say 
that A2{H) is a normal graph. The stronger claim that A2{H) is Dehn convex will 
be proved in the Appendix along with the Normal Form Theorem. As pointed out 
earlier, the diameter d of A2(-ff) can be calculated by Dijkstra's algorithm. 

We now verify that H has finite index if and only if A2{H) is full, that is, there 
are edges labelled by all the generators of G incident at all vertices. The "if" 
direction is clear. For each vertex v, pick a path from the basepoint Oh to v 
and let v be the label on a^. If is any word on the generators of G, since A2{H) 
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is full, we read w along a path starting at Oh and ending at some vertex v. Thus 
w{v)~^ G H and w is on the same right coset of H as v. 

To show that if A2{H) is not full then H has infinite index, suppose that there 
is no edge labelled by a generator a£ at some vertex v. Since A2{H) is a normal 
graph, we can choose a path ay from the basepoint Oh to v whose label is in normal 
form. We want to show that if w is the label on the path ay and a£ is the generator 
missing at v then wa£ is still in normal form. 

Suppose that ay has length at least two. (The special cases where a„ is empty 
or has length one are easily dealt with on the same lines as the general argument.) 
Let ttj be the label on the last edge ei in ay and let Oj be the label on the next to 
last edge 62 in ay. Now i ^ j- If v is primary then i ^ £ because 6162 lies on an 
(i, j)-relator cycle and j is not missing at v. If v is a critical secondary vertex there 
are two possible cases for the path ay as pictured below. 



" a,- 



at 



at 



ai 
->— 



Figure 6 

In both cases there must be an edge incident at v so again £ ^ i. If both 
endpoints of ei are secondary and not critical then ei lies only on an (i,j)-cycle 
and I ^ i. If the vertex on both e\ and 62 is critical and the last vertex of e\ is a 
secondary noncritical vertex then it is possible that £ = i but in this case there is 
an edge labelled by some at., t ^ preceding 62 in ay. 

We can now construct a word z such that for all n > 1, 2;"^ is in normal form 
and ^ H. The existence of such an element shows that H has infinite index. 
We have seen that wai is in normal form and either ai is different from the last 
two generators in w or wai ends in ataiajai and t is different from i and j. Since 
there are at least three generators we can set z = waiartts where differs from 
both a£ and the last generator occurring w and is different from a^. and the first 
generator occurring in w. Then is in normal form for all n > 1 and since A.2{H) 
is a normal graph for H, ^ H for all n > 1. 

The properties of /S.2{H) are heavily used in the proof. Note, however, that if any 
relator cycles are added in Phase II then there are definitely secondary noncritical 
vertices of degree 2 and thus the graph is not full. So the graph is full if and only 
if Phase I perimeter reduction already results in a full graph. 

How to decided membership in H is now clear. Given an arbitrary word w, to 
decide if G -ff, calculate a Dehn reduced form w oi w and then read w on the 
graph A2(-ff) and see if one stays in the graph and returns to the basepoint at the 
end of w. 
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Figure 7 

We now turn to separability. liw^H, we modify A2{H) to Ao{H, w) by adding 
a stem to A2{H) which is a path consisting of new edges labelled by w starting at 
the basepoint Oh and having terminal vertex T^jj. Then we again carry out both 
perimeter reduction and 2-completion to obtain a graph A2{H, w) having the same 
properties as A2{H) but with a second distinguished vertex 7^ Oh and a path 
from Oh to with label w. 

We now show that if G also satisfies the Separability Condition we can directly 
define an action of G on the set V of vertices of A2{H, w) such that Oh = Oh ■ g if 
and only if g E H but — Oh ■ w. The corresponding homomorphism (p thus has 
(p{w) ^ (p{H). Since all generators of G have order 2, there is a natural candidate 
for such an action. Namely, if there is an edge labelled by in A2{H,w) incident 
at a vertex v and the other endpoint of the edge is vi (possibly V2 = v) then 
interchanges v and vi. If there is no edge incident at v with label then fixes 
V. This definition has the property that v — v ■ for all i and all v. 

We now have to check that if and aj are any two distinct generators which are 
related and v is any vertex then v = v - (aittj)'^^^ and we really have an action of the 
Coxeter group G.If there is neither an i-edge nor a j-edge incident at v then and 
ttj both fix so ■ [aittj]^ = v for all k. Suppose first that v is a, primary vertex. 
If there are both an i-edge and a j-edge incident at v, then there is an (i, j)-relator 
cycle beginning at v and thus v ■ (aiaj)'^^^ = v. If there is, say, an i-edge ei incident 
at V but not a j-edge, then there is also not a j-edge incident at the other endpoint 
vi of ei by the properties of A2{H, w). (If there were then ei would again be on an 
(i, j)-relator cycle.) 




Figure 8 

Now V ■ {ttittj)'^ = V since 

V • aiajaittj = vi • ajaiaj = vi ■ a^aj = v ■ aj = v. 

Thus the assumption that m^- is even ensures that v ■ (aittj)'^^^ = v. 
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Similarly, if v is a noncritical secondary vertex, the only two edges incident at v 
lie on a relator cycle. If this is an (i, j)-relator cycle, v-^aittj)"^^^ = v. If there is only 
an z-edge ei incident at v and the other endpoint vi of ei is not a critical vertex, 
then there is again no j-edge incident at vi and the argument of the preceding 
paragraph applies. 

/^2{H, w) has the property that the only kind of vertex at which both an z-edge 
and a j-edge can meet and not lie on a relator cycle is a critical secondary vertex v. 







a, 






at 






aj 


at 



Vl V V2 



Figure 9 

In this case there is also a t-edge, t ^ i,j incident at v and and aj are both 
related to at- (Thus Oi, aj and at lie on a triangle in the modified Coxeter graph F.) 
There is no j-edge incident at vi and there is no z-edge incident at V2- Now we have 
V ■ {ttittj)^ = V. Since aj and aj lie on a common triangle v in F, the Separability 
Condition ensures that rriij is divisible by 3, so v ■ {aiCij)'^^^ = v. The reader can 
easily verify that in this situation we also have vi ■ (aiaj)^ — vi and f 2 • {aiGj)^ — V2, 
which takes care of the last remaining case. 

As pointed out in the introduction, many Coxeter groups are not subgroup sepa- 
rable since there are Coxeter groups with unsolvable membership problems. Proving 
subgroup separability for Coxeter groups which have small exponents seems to re- 
quire a great deal of specific geometric information. See for example [L-R] . 

Perimeter reduction gives a nice proof that the Reduction Hypothesis ensures 
that the Phase I construction of Ai(iy) terminates. Extensive computer investiga- 
tion very strongly suggests, however, that Phase I always terminates for any finitely 
generated subgroup of a Coxeter group of extra-large type but proving this would 
seem to be combinatorially complicated. We believe the following conjecture. 

Conjecture. Theorem I is true as stated for all Coxeter groups of extra-large type. 

We certainly do not have a good feeling for how essential the Separability Con- 
dition is with regards to actual subgroup separability but it is perhaps illuminating 
to consider residual finiteness. It is well-known that every finitely generated Cox- 
eter group G is residually finite since G is a finitely generated subgroup of some 
GL(n, M). In general however, given a presentation and an element w the linear 
group argument does not at all tell us how to actually find a homomorphism from 
G to a finite group with ip{w) ^ I. If G is of extra-large type and satisfies the 
Separabilty Condition we have the following. 



13 



Theorem III. There is a uniform quadratic time algorithm which, when given any 
Coxeter group G of extra-large type satisfying the Separability Condition and any 
element w ^ 1 of G. writes out a homomorphism (f) from G to a finite symmetric 
group S such that (/){w) 7^ 1. 

Proof Given a nontrivial element of G, let whe a Dehn reduced form. Let Ai{w) 
be the graph consisting of a path of successive edges labelled by the letters in w. 
Since w ^ 1, the initial vertex O^y and the terminal vertex of Ai{w) are distinct. 
Since w is Dehn reduced it does not contain more than half of a defining relator and 
thus Ai{w) has the Relator Path Property that there are no (z,j)-paths requiring 
fewer than four edges to be completed to a relator cycle. We can then construct 
A2{w) from Ai{w) as before. Since G satisfies the Separability Condition , we can 
define the action of G on the set v of vertices of A2{w) as before and w sends Ow 
to so w is not in the kernel of the action. □ 

Recall that Peter Scott [SC] proved that all Fuchsian groups are subgroup separa- 
ble by showing that surface groups are subgroup separable and these are separable 
because an appropriate Coxeter group is subgroup separable. Scott showed that 
subgroup separability is a virtual property: If [K : G] < 00 and K is subgroup 
separable then G is subgroup separable. So any finitely generated group containing 
a subgroup separable Coxeter group of finite index is itself subgroup separable. 

§3. Surface Groups. 

McCammond and Wise pointed out that surface groups form an important class 
of groups to which we can apply perimeter reduction. We point out here that 
constructing the 2-completion of a subgraph graph works for surface groups in 
exactly the same way as for Coxeter groups. The 2-completion therefore gives all 
the detailed information of our Theorem except for subgroup separability. The 
Reduction Hypothesis we need is that the standard defining relator has length at 
least 8, that is, the genus is at least 2 in the orientablc case and is at least 4 in the 
nonorientable case. As usual, the symmetrized set R of defining relators consists of 
all cyclic permutations of the standard relator and its inverse. 

We briefly discuss the construction. Given a flnite set {hi, . . . , hm} of generators 
of a subgroup H oi a surface group G satisfying the Reduction Hypothesis, we begin 
with the bouquet of circles Ao{H) as always. Now, of course, the generators have 
infinite order. 

Phase I now consists of completing paths which are labelled by more than half of 
a defining relator to relator cycles. Let h be half the length of the defining relator. 
The count 7(A) for a graph A is the sum over all edges e of A of /i times the number 
of cycles missing at e plus the total number of edges. 

Suppose that we complete a path a of length £ > h to a relator cycle. We add 
k < h new edges. Because the defining relator is quadratic each generator begins 
only two distinct relator cycles. Thus we have removed £ missing cycles and added 
at most k missing cycles and the count is reduced. So completing paths containing 

14 



Figure 10 



more than half of a defining relator and folding terminates in a linear number of 
steps by perimeter reduction, yielding the graph Ai{H). As in the case of Coxeter 
groups, reading the two initial letters of an element r of the symmetrized set R 
uniquely determines r and any path not already part of a relator cycle requires at 
least four edges to complete. The construction of the 2-completion A2{H) thus 
proceeds exactly as in the case of Coxeter groups. 

As before, A2{H) is the transition graph of a finite automaton which accepts 
a Dehn reduced word w if and only if w E H and the diameter of A2{H) is a 
quasiconvexity constant for H. Also as before, H is of finite index if and only if 
A2{H) is a full graph. 

Pittet[P] earlier showed that fundamental groups of orientable surfaces of genus 
at least two are locally quasiconvex. The argument of McCammond and Wise [M- 
W] and our argument here show that such surface groups are locally quasiconvex 
and have uniform membership problem solvable in quadratic time. In his thesis [G] 
Grunschlag proved the following theorem. 

Theorem (Grunschlag). If G is a finitely generated group whose uniform member- 
ship problem is solvable in time f{n), and J contains G as a subgroup a subgroup of 
finite index, then the uniform membership problem for J is solvable in time f^{n). 

In particular, this theorem shows that having uniform membership problem solv- 
able in polynomial time is a virtual property. Also, several authors, see for example 
Kapovich and Short [KS2], have shown that local quasiconvexity is a virtual prop- 
erty. 

Theorem IV. // F is any group containing as a subgroup of finite index either 
an orientable surface group of genus at least two or a Coxeter group which satisfie 
the Reduction Hypothesis then F is locally quasiconvex and the uniform membership 
problem for F is solvable in at most quartic time. 

In particular, the theorem applies to finitely generated discrete groups of isome- 
trics of the hyperbolic plane. It seems to the author that having uniform member- 
ship problem solvable in polynomial time is an interesting property which may help 
point out which groups have "really good" geometry. 
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§4. Appendix 
Normal Forms and Dehn Convexity 

In this appendix we sketch a proof of the facts we used about normal forms and 
geodesies. Let G = {X;R) be a finitely presented group where i? is a symmetrized 
set of relators satisfying the small cancellation condition C (^) (See [L-S] for the 
basic definitions of small cancellation theory.) For convenience we suppose that all 
pieces have length 1 since this is the case in the groups which we have considered. 
We want to prove that the subgroup graphs A2{H) which we have constructed are 
indeed Dehn convex in the sense that if there is a path a between two vertices 
labelled by a reduced word w and ^ is a Dehn reduced word with z = w in G then 
there is a path labelled by z between the two vertices. 

Our A2{H) all have the property that any path 7 labelled by a subword of an 
element of R containing at least three letters is part of a relator cycle. So let F be a 
finite graph over the group G which has this property. We shall show that F is Dehn 
convex. Let w be the label on a reduced path between two fixed vertices. Certainly, 
if w contains a subword s which is more than half a defining relator, the part of w 
labelled by s lies in a relator cycle and we can perform a Dehn reduction by going 
around the shorter part of the relator cycle and obtain a shorter path between the 
two vertices. Iterating this procedure, we may assume that w is i?-reduced. 

Let z be the shortlex normal form of w to show that there is a path in F labelled 
by z between the two vertices. We consider the equality diagram E{w, z) of w and 
2;, which is formed as follows. Write w = pw' and z = pz' in the free group where p 
is the maximal common initial segment of w and z. The prefix p may be empty but 
if either w' or z' is empty then w = z is the free group. (If we had z' empty and 
\w'\ > we would have a nontrivial i?-reduced word w' equal to 1 in G which is 
impossible.) Now write w = pwid, z = pzid where d is the largest common terminal 
segment of w' and z' . Again, neither wi nor zi are empty unless w = z and is thus 
already in normal form and we arc done. 

Consider the minimal reduced diagram G for the nontrivial cyclically reduced 
word wiZi^. The equality diagram E{w, z) is the reduced -R-diagram 




z 



Figure 11 

which reads w along the top boundary a and z along the bottom boundary r. (So a 
boimdary cycle oiE{w,z) is err ~^.) Equality diagrams have been often used in small 
cancellation theory and have a very simple structure which seems to first appear 
in Schupp's [SI] analysis of the structure of conjugary diagrams for certain small 
cancellation groups. Since no region can have more than half of its boundary as a 
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consecutive part of either cr or r, successive applications of Greendlinger's Lemma 

show that each region D has interior degree at most 2 and the intersection of dD 
with the top (bottom) boundary of E{w, z) is connected. Thus E{w^ z) of some 
number of "islands" connected by segments or meeting at a single vertex. 




Figure 12 



In each island, a region has interior degree if it is the entire island, interior 
degree 1 if it is at the end of an island and interior degree 2 otherwise. Since both 
w and z are i?-reduced and R satisfies C" (y), for each region D, the segment which 
comprises the intersection of dDCicr (respectively dDdr) must contain at least two 
letters and is thus labelled by a nonpiece. 

A vertex of a and a vertex of r which are connected by an interior edge are said 
to correspond. A vertex in cr fl r corresponds to itself. Now the word w is both 
the label on the path a in F between vertices Vq and Vn and the label on the top 
boundary a of the equality diagram E{w, z). Working from left to right in E{w^ z) 
we can successively show that for each initial segment u oiw which ends at a vertex 
Si corresponding to a vertex ti e r, there is a path (3i in F between vq and the vertex 
Vi at which ends in F which has the same label as the path in E{w, z) obtained 
by starting at its initial vertex sq, reading along r until the vertex ti corresponding 
to Si is encountered and then going along the interior edge from ti to Sj. (The latter 
provided ti ^ Si.) The picture is 




h H 



Figure 13 

Suppose, for example, that E{w,z) begins exactly as in the figure above. The 
word ui must contain at least three letters of the relator cycle labelling the region 
Di. Thus the subpath of a labelled by ui lies on a relator cycle in the graph F 
and going around that cycle the other way gives the desired path Pi in F from vq 
to vertex Vi. The only point to notice is that in any event the label on a segment 
dDi n a contains at least two letters. Thus the path in F starting with the last 
edge in (3i and then following that part of a labelled by S2 must contain at least 
three letters. This path thus is also part of a relator cycle and we can go around 
the other way. Following /?i by this path and omitting the edge traversed twice we 
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obtain a reduced path P2 in F from vq to the vertex 1/2 labelled by the desired 
word obtained by reading along r from to t2 and then taking the interior edge 
to S2. We can clearly continue in this way, finally obtaining the desired reduced 
path (3 with label z which goes from Vo to Vn in F. 

We have thus shown that if w is the label on a path in F between two vertices 
then there is a path between these vertices whose label is the normal form z of w. 
Now let w' be any /^-reduced word equal to z in the group G. We now take the 
equality diagram E{w', z) and work the other way. We have the path labelled by z 
existing in F. Apply the argument above but work from r to cj to construct a path 
labelled by w' in F. In particular, this shows that F is Dehn convex in the sense 
that if w is a word labelling a path a in F and u is any Dehn reduced word equal 
to 10 in G then there is a path labelled by u between the endpoints of a. 

The argument of working from left to right in an equality diagram also shows 
that the set of normal forms of elements of G is a regular language. First of all, 
the set of all Dehn reduced words is certainly regular since it is defined by not 
containing a subword from a fixed finite list. Our automaton verifies that an input 
w is Dehn reduced and as long as it keeps track only of one possible region in an 
equality diagram plus two pieces of information. For example, suppose that G is 
the Coxeter group 

(^=(01,02,03; a-, (oiOjy, ij^j) 

The point is that at any point in a Dehn reduced word w the possible corre- 
sponding part of the other boundary r of E{w,z), z a normal form, is completely 
determined. Suppose, for example, that w = a2aia2aia3a2a3aia2aia2as. When 
two consecutive letters do not form part of a region, we start a new region and we 
start a new region at the beginning of w. Once two letters are part of a relator, that 
relator is uniquely determined and thus the label in the bottom part. On reading 
0201 we know that a first region starts 




Figure 14 

When we arrive at a break in w of the current relator, we know how to complete 
the first region. In our example we have 

The two pieces of information which the automaton needs to remember is whether 
the proposed label for the normal form is lexicographically less than the word being 
read, in this example it is, and the difference in length between the top and the 
bottom. Notice that this length cannot be greater than 1. If it is, we can find a 
shorter word equal to w by going around the bottom and then traversing the interior 
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Figure 15 

edge up to the corresponding vertex of w. In the example, when we read 03, we 
switch to a new region and at the end of reading the segment 030203 we have the 
picture 



03 



03 
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02 
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The bottom remains 1 letter shorter than the top. On reading the last letter 03, 
we have the picture 




Figure 17 



The top and bottom now haved the same length but the bottom label comes first 
lexicographically so w is rejected at this point. Since there are only finitely many 
"region pictures" we can use them as the states of a finite automaton and we are 
done. Technically we need a special end marker symbol to indicate that we have 
come to the end of the word, but if the language with the special endmarker is 
regular then so is the language without the endmarker by standard closure properties 
of regular languages. 
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